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(54) Agent interface device 



(57) An agent interface device for providing informa- 
tion suited for user preferences on the basis of the user 
usage history. The agent interface device comprises an 
agent display control means (1 09) for controlling th e dis- 
play and action of the agent, a voice output means (110) 
for generating a speech of the agent, an input means 
(111) for inputting information from the user, an action 
script execution engine (1 01) for driving the agent inter- 
face into action according to an action script describing 
means to control the agent interface, a retrieval engine 
(102) for retrieving an adapted action script from an ac- 
tion script database DB (103) and transferring it to the 



action script execution engine, an application program 
interface (1 05) for controlling an application program, an 
electronic program guide (EPG) information acquiring 
means (106) for obtaining an electronic broadcast pro- 
gram guide, a user preference database DB (108) for 
analyzing preferences of the user from information pro- 
vided by the EPG information acquiring means and the 
application program interface and storing analysis re- 
sults therein, an information providing engine (107) for 
providing information to the user according to informa- 
tion stored in the preference database, and a timer (1 1 2) 
for setting a time for providing information to the user by 
the information providing engine. 
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Description 

BACKGROUND OF THE INVENTION 

[0001] The present invention relates to an agent in- 
terface device using an artificial agent. 
[0002] Japanese Laid-Open Patent Publication No. 
8-305531 discloses a prior art technique by which a per- 
sonified agent to be displayed on a personal computer 
of home electronics is interposed between a control ap- 
plication program of the home electronic apparatus and 
a user so that a user may control the electronic appara- 
tus by or via the agent. 

[0003] However, the prior art involves the following 

problems. 

[0004] The technique disclosed in Japanese Laid- 
open Patent Publication No. 8-305531 provides merely 
the same functions as those of a remote control of a TV 
set when viewing a TV program on a display of a per- 
sonal computer, so a user may loss a chance of viewing 
an interesting TV program if he or she does not positive- 
ly drive in time a television application program. 
[0005] The user must positively acquire TV program 
guides to know a schedule of broadcasting interesting 
TV programs. 

[0006] With the prior art, the user may not be aware 
if the application device was left as connected to the In- 
ternet for a long time. 

[0007] With the prior art, the user cannot know wheth- 
er resources become short when using a plurality of ap- 
plications. 

SUiVIIVIARY OF THE INVENTION 

[0008] The present invention was made to solve the 
above problems involved In the prior art device. 
[0009] Accordingly, the present invention is intended 
to provide an agent capable of teaching a user a TV pro- 
gram recommendable or suitable to the user preference 
a little while before the beginning of the program not to 
loss a chance of viewing it. 

[0010] The present invention is also intended to pro- 
vide an agent capable of providing a guide and informa- 
tion of programs suited to the user preference. 
[0011] The present invention is further intended to 
provide an agent capable of advising the user to cut off 
the device connected to the Intemet for a long time. 
[0012] The present invention is still further intended 
to provide an agent capable of informing the user of 
shortage of the system resource to prevent the system 
from unstably operating. 

[001 3] An object of the present invention is to provide 
an agent interface device for displaying an artificial 
agent for providing inf onnation by the agenVs action and 
speech, comprising an agent interface having an agent 
display control means for controlling the display and ac- 
tion of the agent, a voice output means for generating a 
speech of the agent, and an input means for input by a 



user, an action script execution engine for driving the 
agent interface into actual action according to an action 
script describing means to control the agent interface, 
a retrieval engine for retrieving an adapted action script 

5 from an action script database and transferring the ac- 
tion script to the action script executing engine, an ap- 
plication program interface for controlling an application 
program, an electronic program guide information ac- 
quiring means for acquiring an electronic broadcast pro- 
gram guide, a preference database for analyzing user 
preferences from information provided by the electronic 
program guide infonrtatlon acquiring means and the ap- 
plication program interface and storing analysis results 
therein, an Information providing engine for providing in- 

^5 formatioji to a user according to infomnation stored in 
the preference database, and a timer for setting a time 
for providing infomiatlon to the user by the infomiatlon 
providing engine. 

[0014] Another object of the present invention is to 

20 provide an agent interface device, characterized in that 
it is further provided with means capable of deciding a 
user preferred television program from a user usage his- 
tory of a television application controllable by the appli- 
cation program interface according to infomnation from 
the electronic program guide information acquiring 
means and information from the preference database, 
of displaying the agent at a specified time (n minute) 
before a start tim e of the user preferred TV program and. 
of causing the agent to have a conversation with the us- 

3o e r abo ut th e user preferred TV program by a speech out- 
put through the voice outputting means. 
[0015] Another object of the present invention is to 
provide agent interface device, characterized in that it 
is further provided with means capable of deciding a us- 

35 er preferred genre from a user usage history of a tele- 
vision application contrcllable by the application pro- 
gram interface according to infomiatlon from the elec- 
tronic program guide information acquiring means and 
information from the preference database, of displaying 

40 the agent at a specified time (n minutes) before a start 
time of the user preferred genre TV program, and of 
causing the agent to have a conversation with the user 
about the user preferred genre by a speech through the 
voice outputting means. 

45 [0016] Another object of the present Invention is to 
provide agent interface device, characterized in that it 
is further provided with means capable of displaying the 
agent Immediately after the end of a TV program of a 
television application controllable by the application pro- 

so gram interface and of causing the agent to have a con- 
versation with the user about the just viewed program. 
[0017] Another object of the present invention Is to 
provide agent interface device, characterized in that the 
agent is displayed Immediately after ending a television 

55 application controllable by the application program in- 
terface and an intemet retrieval means for retrieving tn- 
fomiation on the Internet is provided to retrieve informa- 
tion from a web site relevant to the just viewed program. 
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[0018] Another object of the present invention is to 
provide agent interface device, characterized in that it 
is further provided with means capable of displaying the 
agent immediately after ending a video recording device 
controllable by the application program Interface and of 
making conversation with the user about a just viewed 
video program. 

[0019] Another object of the present invention is to 
provide agent interface device, characterized in that the 
agent is displayed immediately after ending a video re- 
cording device controllable by the application program 
interface and an internet retrieval means for retrieving 
information on the Internet is further provided to retrieve 
infomnation from a web site relevant to a just viewed vid- 
eo program. 

[0020] Another object of the present invention is to 
provide agent interface device, characterized in that it 
is provided with means capable of displaying the agent 
immediately after ending an internet browser application 
program controllable by the application program inter- 
face arid of causing the agent to make conversation with 
the user about a just browsed web site. 
[0021] Another object of the present invention is to 
provide agent interface device, characterized in that the 
agent is displayed immediately after ending an internet 
browser application program controllable by the appli- 
cation program interface and an internet retrieval means 
for retrieving information from a web site on an internet 
is provided to browse a web site relevant to a just 
browsed site. 

[0022] Another object of the present invention is to 
provide agent interface device, characterized in that it 
is capable of changing a theme of a conversation in ac- 
cordance with a keyword of a program displayed by a 
television application controllable by the application pro- 
gram interface. 

[0023] Another object of the present invention is to 
provide agent interface device, characterized in that it 
is capable of changing a theme of a conversation ac- 
cording to time interval information obtainable from a us- 
age history of a television application controllable by the 
application program interface. 

[0024] Another object of the present invention is to 
provide agent interface device, characterized in that a 
table of frequently watched genre programs is prepared 
on the basis of a usage history of a television application 
controllable by the application program interface and 
displayed when driving a television application control- 
lable by the application program interface. 
[0025] Another object of the present invention is to 
provide agent interface device, characterized in that the 
agent is capable of informing the user of a communica- 
tion line still connected to the Internet when n minutes 
elapsed after driving an intemet browser application 
program controllable by the application program inter- 
face. 

[0026] Another object of the present invention is to 
provide agent interface device, characterized in that the 



agent is capable of informing the user of shortage of a 
system resource when a resource of the system runs 
short (it becomes n %). 

s BRIEF DESCRIPTION OF DRAWINGS 

[0027] Fig. 1 is a construction view of an agent inter- 
face device according to the present invention. 
[0028] Fig. 2 is a module construction of an agent In- 
10 terface device according to the present invention. 

[0029] Fig. 3 is an exemplary display image of an 
agent of an agent interface device according to an em- 
bodiment of the present invention. 
[0030] Fig. 4 illustrates an exemplary scene of usage 
^5 of an agent interface device according to an embodi- 
ment of the present invention. 

[0031] Fig. 5 illustrates an exemplary scene of an 
agent interface device according to an embodiment of 
the present invention. 

[0032] Rg. 6 is a flowchart depicting the operation of 
an agent interface device according to an embodiment 
of the present invention. 

[0033] Fig. 7 is an exemplary display image of an 
agent of an agent interface device according to an em- 
bodiment of the present invention. 
[0034] Fig. 8 is a flowchart depicting the operation of 
an agent Interface device according to an embodiment 
of the present invention. 

[0035] Fig. 9 is an exemplary display image of an 
agent of an agent interface device according to an em- 
bodiment of the present invention. 
[0036] Fig. 1 0 Illustrates an exemplary scene of usage 
of an agent interface device according to an embodi- 
ment of the present invention. 

[0037] Fig. 1 1 illustrates an exemplary scene of usage 
of an agent interface device according to an embodi- 
ment of the present invention. 

[0038] Fig. 12 is a flowchart depicting the operation 
of an agent interface device according to an embodi- 
ment of the present invention. 

[0039] Fig. 13 is a flowchart depicting the operation 
of an agent interface device according to an embodi- 
ment of the present invention. 

[0040] Fig. 14 is an exemplary display image of an 
agent of an agent interface device. according to an em- 
bodiment of the present invention. 
[0041] Fig. 15 is a flowchart depicting the operation 
of an agent interface device according to an embodi- 
ment of the present invention. 

[0042] Fig. 16 is an exemplary display image of an 
agent of an agent interface device according to an enn- 
bodiment of the present invention. 
[0043] Fig. 17 is a flowchart depicting the operation 
of an agent interface device according to an embodi- 
ment of the present invention. 

[0044] Fig. 18 is an exemplary display image of an 
agent of an agent interface device according to an em- 
bodiment of the present invention. 
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[0045] Fig. 19 is a flowchart depicting the operation 
of an agent interface device according to an embodi- 
ment of the present Invention. 

[0046] Fig. 20 is an exemplary display image of an 
agent of an agent interface device according to an em- 
bodiment of the present invention. 
[0047] Fig. 21 is a flowchart depicting the operation 
of an agent interface device according to an embodi- 
ment of the present invention. 

PREFERRED EMBODIMENT OF THE INVENTION 

[0048] A preferred embodiment of the present Inven- 
tion will be described bellow with reference to the ac- 
compEinylng drawings. 

[First embodiment] 

[0049] Fig. 1 is a construction view of an agent inter- 
face device according to the present invention. 
[0050] A CPU 1 0. a memory 1 1 , a hard disk 1 2, a FD 
(Floppy Disk) drive portion 13, a communk^ation portion 
14, an optical disk drive 15, a TV tuner board 16, a dis- 
play portion 20, a keyboard and mouse (pointing device) 
22 are connected to a bus 200. A microphone 1 8 and a 
speaker 1 9 are connected through a sound mixer 1 7 to 
the optical disk drive 15 and the TV tuner board 16 in 
order for the user to have a conversation with an agent 
of a current application. The communication device 14 
Is connected to the Internet 23 to communicate with in- 
formation sources on the Internet. 
[0051] Fig. 2 shows an example of a function module 
construction of an agent interface device according to 
the present invention. The agent interface device com- 
prises an agent interface 1 00, an action script execution 
engine 1 01 , a retrieval engine 1 02, an action scripts da- 
tabase 1 03, a dictionary 1 04, an application program in- 
terface 105, an electronic program guide (ERG) infor- 
mation acquisition means 106, an information providing 
engine 107, a user preference database DBIOS and a 
timer112. An agent display control means 109 is a mod- 
ule used for presenting an artificial agent (a personified 
agent or an Impersonated agent) on a display portion 
20 and controlling its actions, a speech output means 
(voice output means) 110 is a module for providing the 
agent with voice and outputs a speech of the agent 
through the speaker 1 9. The speech may be a synthe- 
sized voice speech or recorded human voice speech. It 
is not restricted to any of them. The input means 111 is 
a module for inputting a user's request to the agent by 
using a microphone 18, keyboard 21 or mouse (pointing 
device) 22, The Input module 111 includes a voice rec- 
ognition function for recognizing a speech inputted 
through the microphone 18. The agent interface 100 is 
composed of the agent display control means 109^ the 
speech output means 110 and the input means 111. It 
serves as an interlace between the user and the appli- 
cation program interface 105. This application program 



Interface 1 05 Is an interface that controls an application 
program and a control application program for a home 
electronic apparatus. The applfcation program interface 
1 05 controls a variety of applications on a personal com- 
s puter and control applications for home electronic appa- 
ratuses such as televisions, video recorders and optical 
disk record players. The action script database 1 03 con- 
tains action scripts each of which describes a procedure 
of realizing a particular function by co-operation of the 
agent interface 100 and the application program inter- 
face 105. An example of an action script is as follows: 
The agent Infomns the user of that a recommendable TV 
program suited to the user preference will start n min- 
utes later. 

*5 [0052] The action script execution engine 101 analyz- 
es an actual action script and controls the agent inter- 
face 100 and the application program interface 105 so 
that they may cooperatively execute the designated 
function. The retrieval engine 102 is a module that re- 

20 trieves an adaptive script from the action script database 
DB103 according to a retrieval request from the input 
means 111 and transfers the action script to the action 
script execution engine 101 . Since the retrieval request 
inputted vocally through the input means 111 includes a 

25 natural language, the retrieval engine 102 refers to the 
dictionary 104 If necessary, retrieves an action script 
most suited to the request, for example, an action script 
for "driving a television application program and select- 
ing channel 2" from the action script database 103 and 

30 transfers it to the action script execution engine 101. The 
EPG information acquisition means 106 is used for ac- 
quiring an electronic television program guide (EPG). It 
may obtain the EPG, for example, from a web page for 
providing TV program infonmation by accessing It 

35 through the communication device 14 connected to the 
Internet 23 or may acquire the EPG from a data broad- 
cast by using the TV tuner board 16. The information 
providing engine 1 07 is used for storing data in the pref- 
erence database DB108 and generating recommenda- 

40 tlon to the user based on information accumulated In the 
preference database DB108. 

[0053] In case of TV programs, the infonmation pro- 
viding engine 1 07 obtains a TV state from the television 
application program interface which is one of the appli- 

45 cation program interface 1 05, refers to a program guide 
obtained from the EPG acquiring means 106 and 
records data of the program viewed by the user into the 
user preference database DB108. Based on the user 
usage history in the user preference database DB108, 

50 the information providing engine 1 07 refers to a table of 
programs obtained from the EPG acquiring means 1 06 
to be broadcast, acquires data of programs that the user 
may want to watch and transfers the data to the action 
script execution engine 101. In this case, the informa- 

55 tion-providing engine may use the retrieval engine 1 02 
if necessary. 
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[Second Embodiment] 

[0054] Fig. 3 shows the first example of a screen im- 
age of an agent, according to the second embodiment 
of the present invention. Based on information on pro< 5 
grams watched by the user and registered in the user 
preference database DB1 08, the agent interface device 
refers to a TV program guide acquired by the EPG in- 
formation acquisition means 106, finds therein a pro- 
gram to be preferably viewed by the user and timely io 
transfers the selected program information to the action 
script execution engine 101 . By the effect of the timer 
112, an. agent is displayed on a display screen "n" min- 
utes before the beginning the program to inform the user 
that a recommended TV program will start n-minutes lat- ^ 5 
er. For example, let assume that the user often viewed 
soccer game broadcasts on the TV set. In this case, the 
agent interface device stores a user keyword "soccer 
game broadcast" in the user preference database OB, 
refers to a TV program guide obtained by the EPG ac- 20 
quiring means 106 to find therein a soccer game to be 
broadcast and obtain information including date, start 
time and channel of the broadcast. As soon as it reached 
"n" minutes before the beginning of the broadcast on the 
date, the agent interface device displays on the TV dis- 25 
play an agent to inform the user that the broadcast of a 
soccer game will start soon. The time to provide infor- 
mation can be preset on the timer 112. Now let assume 
the time is set to 5 minutes. In this instance, the agent 
appears on the TV screen 5 minutes before the begin- 30 
ning of the broadcast and, for example, speaks to the 
user "your favorite soccer game will be soon on air. Do 
you want to see it?" and at the same time presents the 
same text in a window on the TV screen. When the user 
vocally answered "YES" (as shown in Fig. 4) or inputted 35 
a response "Yes" by using the keyboard 21 or mouse 
(pointing device) 22, the television application program 
Is activated and indicates the channel 2 on the TV 
screen as shown in Fig. 5. 

[0055] Fig. 6 is a flowchart depicting the operation of 40 
the information providing engine 1 07 of the agent inter- 
face device. When the application program started 
(Step S61 ), the information providing engine 1 07 exam- 
ines the state of the television application program at 
specified intervals (Step S62) and, if no TV program is 
displayed on the TV set (Step S63), proceeds to Step 
S66. If the TV program Is viewed in Step S63, the infor- 
mation providing engine 107 identifies the program that 
the user now enjoys, referring to an EPG (Electronic 
Program Guide) (Step S64) and. then, stores the TV so 
program data into the user preference database DB108 
(Step S65). The EPG is obtained by the EPG informa- 
tion acquisition means 1 06. In practice, the EPG infor- 
mation is available from the Internet or data broadcast- 
ing by any of usual methods not defined herein. In Step ss 
S66, the information providing engine 107 examines 
whether there is a request to present recommendable 
programs. If no request is given (Step S67), the engine 



1 07 returns to Step S62. If any request was given (Step 
S67). the engine DB1 08 refers to the usage history con- 
tained in the user preference database DB1 08 to extract 
a keyword common to the programs that the user pre- 
ferred specially (Step S68). In this case, the engine 1 07 
may use the retrieval engine 102 If necessary. In Step 
S69, the engine 107 refers to the EPG of TV program 
to be broadcast and searches therein a program having 
the common keyword previously extracted from the da- 
tabase DB108 or a similarly meaning keyword. In this 
case, the engine 107 may uses the retrieval engine 1 02 
if necessary. In Step S70, the engine 107 extracts the 
date and the start time of the program from the retrieved 
program information and examines by using the timer 
112 whether It is now n-minutes before the beginning of 
the program. If it is not n-minutes before the start time, 
the engine 107 retums to Step S70. If It Is n-mlnutes 
before the start time, the engine 1 07 transfer the extract- 
ed program information to the action script execution en- 
gine 101 that is a origin of requesting for the information. 
The time-to-present may be optionally set by the user. 
The engine 1 07 returns to Step S62. The action script 
to be executed in this instance describes to present an 
agent who has a suitable conversation with the user in 
regard to the selected TV program. 
[0056] The above operation procedure causes the 
agent to appear on a TV screen "n" minutes before the 
start time of the program and have a conversation with 
the user In regard to the recommended TV program. 

[Third Embodiment] 

[0057] In this embodiment, the genre-based recom- 
mendation is used instead of the program-based rec- 
ommendation described for the second embodiment. 
The term "genre" means a category of TV programs ac- 
cording to their contents described in program Informa- 
tion acquired by the EPG acquiring means 1 06. Genres 
(categories) of TV programs are, for example, TV dra- 
ma, movie, sports, entertainment, hobbies/Lrving News, 
music, variety shows, social news and reports, educa- 
tion and animation/puppet shows, etc. The genres will 
not be further defined herein. 

[0058] The operation procedure similar to that of the 
second embodiment is executed to select genre-based 
recommendable programs, display an agent on a 
screen n-minutes before the start time of each of the 
genre-based recommended programs and make con- 
versation with the user in respect to the recommended 
genre-based program. 

[Fourth Embodiment] 

[0059] Fig. 7 shows the second example of a screen 
Image of an agent, according to the fourth embodiment 
of the present invention. The agent interface device of 
this embodiment refers to a TV program guide acquired 
by the EPG information acquisition means 106 and ob- 
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tains therefrom information on a TV program that the 
user is now enjoying. When the user ended the televi- 
sion application program, the agent interface device 
transfers information to the action script execution en- 
gine 101 and causes the same engine to present on a 
TV screen an agent who may have a conversation with 
the user in respect to the TV program that the user 
viewed just before endin g the Tel evision appi icatlon pro- 
gram. For example, let assume the user watched a soc- 
cer game broadcast and just finished the television ap- 
plication program, in this case, the agent immediately 
appears on the TV screen and speaks "The live game 
in a stadium might be more exciting, don't you think so?" 
with the same text indicated in a window. 
[0060] Fig. 8 is a flowchart depicting the operation of 
the information providing engine 1 07 of the agent inter- 
face device. When the program started (Step S81), the 
information providing engine 1 07 examines the state of 
the television application program at specified intervals. 
(Step S82) and returns to Step SB2 if no TV program is 
displayed on the TV set (Step S83). If the TV program 
is viewed in Step S83, the engine 107 refers to an EPG 
and identifies the program that the user now enjoys 
(Step S84). The engine 107 then extracts the TV pro- 
gram infomnation from the EPG, which information in 
this instance may contain the program name and the 
genre name. In Step S86, the engine 107 examines 
whether the television application program ended, and, 
if not, returns to Step S84. If the program ended (Step 
S86), the engine 107 transfers the extracted action 
script (TV program information) to the action script ex- 
ecution engine 1 01 (Step S87) and then returns to Step 
SB2. The action script describes that an agent is pre- 
sented to make conversation with the user in respect to 
the program infomnation. 

[0061] The above operation of the embodiment can 
realize such a particular effect that, as soon as the TV 
program ended, the agent appears on the TV screen to 
have a conversation with the user in respect to the Just 
ended program. 

[Fifth Embodiment] 

[0062] Fig. 9 shows the third example of a screen im- 
age of an agent, according to the fifth embodiment of 
the present invention. This embodiment refers to a TV 
program guide acquired by the EPG Information acqui- 
sition means 106 and obtains therefrom information on 
a TV program that the user is now viewing. When the 
user ended the television applrcatron program, the agent 
interface device transfers Information to the action script 
execution engine 101 to present an agent on a TV 
screen, then connects to the Internet 23 through the 
communication portion 14 and searches an information 
site relating to the program just viewed by the user For 
example, it is assumed the user watched a soccer game 
broadcast and just ended the Television application pro- 
gram. In this instance, the agent immediately appears 



on the TV screen and speaks "How about searching an 
Internet web page relating to <your favorite soccer>?" 
with the same text indicated in a window. When the user 
vocally answered "Yes" or inputted the same response 
5 by using the keyboard 21 orthe mouse (pointing device) 
22, the agent interface device drives an application pro- 
gram to carry out browsing a "soccer" web site on the 
Internet. 

[0063] Fig. 12 is a flowchart depicting the operation 

10 of the infomnation providing engine 1 07 of the agent in- 
terface device. When the program started (Step SI 21 ). 
the information providing engine 1 07 examines the cur- 
rent state of the television application program at spec- 
ified intervals (Step SI 22) and returns to Step S 122 if it 

15 judged no TV program being viewed on the TV set (Step 
S1 23). Having judged a TV program being viewed (Step 
SI 23), the engine 107 refers to an EPG and identifies 
the program that the user now enjoys (Step SI 24). The 
engine 107 then extracts a keyword, e.g., the TV pro- 

20 gram title (Step S125). In Step SI 26, the engine 107 
examines whether the user ended the television appli- 
cation program. If the television application program is 
not ended, the engine 107 returns to Step SI 24. When 
the television application program ended (at Step 

2S S126), the engine 107 transfers an action script to the 
extracted keyword to the action script execution engine 
101 (Step SI 27) and then returns to Step SI 22. The 
action script describes that an Internet browser applica- 
tion program shall be driven to search for the keyword 

30 and browse relevant web sites. 

[0064] The above operation of this embodiment real- 
izes such a particular effect that web site pageis relating 
to the TV program just watched by the user Is explored 
and browsed as soon as the TV program ended. 

35 

[Sixth embodiment! 

[0065] This embodiment is analogous to the fourth 
embodiment but deals with video programs recorded on 

40 the recording devices (media) such as a video recording 
tape, CD-RW, DVD-RW, MO and hard disk, while the 
latter deals with TV programs viewable by the television 
application. Therefore, the agent interface device In this 
embodiment recognizes the end of a video program (i. 

45 e., a recording device) and transfers infomnation on the 
video program to the action script execution engine 101 . 
[0066] Based on the video program information, this 
embodiment con ducts the same procedure as that of the 
fourth embodiment to display an agent immediately after 

so the end of the video program and have a conversation 
with the user in regard to the just watched program. 

[Seventh Embodiment] 

55 [0067] This embodiment is analogous to the fifth em- 
bodiment but deals with video programs recorded on the 
recording devices (media) such as a video recording 
tape, CD-RW, DVD-RW, MO and hard disk, while the 
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laner deals with TV programs viewable by the television 
application. Therefore, the agent interface device tn this 
embodiment recognizes the end of a video program (i. 
e., a recording device) and transfers information on the 
video program to the action script execution engine 101. 
[0068] Based on this infonmation, this embodiment 
displays an agent immediately after the end of the video 
program and starts searching a web site relating to the 
just watched program. 

[Eighth Embodiment] 

[0069] In this embodiment, the television application 
described in the fourth embodiment is replaced by an 
Internet browser application program and infomiation of 
an browsed web site is transferred to the script execu- 
tion engine 101 after recognition of finishing the Intemet 
browser application program. 

[0070] The operation of the embodiment realizes 
such a particular effect that an agent appears on a dis- 
play screen immediately after the end of the Internet 
browser application program to have a conversation 
with the user in regard to the just viewed web site. 

[Ninth Embodiment] 

[0071] In this embodiment, the television application 
described in the fifth embodiment Is replaced by an In- 
ternet browser application program and infonnation of 
an browsed web site is transferred to the script execu- 
tion engine 101 after recognition of finishing the Internet 
browser application program. 

[0072] The operation of the embodiment realizes that 
an agent appears on a display screen Immediately after 
the end of the Internet browser application program and 
starts browsing another web site relating the just viewed 
web site in the Internet. 

[Tenth Embodiment] 

[0073] In this embodiment, an agent can change the 
theme of a conversation with the user depending on a 
keyword of a TV program being viewed by the user For 
example, when the user did an incorrect operation with 
watching no TV program, an agent appears on a display 
screen and outputs a message "I cannot accept your 
operation" in both in the speech mode and the text 
mode. For example, if a keyword "soccer" is extracted 
while watching the soccer game broadcast, the agent 
Eippeans and outputs a spoken message and a same 
text message regarding the extracted keyword, e.g., "I 
cannot acceptyour request. You may receh/e a Red card 
If you should repeat the incorrect operation so may 
times". 

[0074] Fig. 13 is a flowchart depicting the operation 
of the infomiation providing engine 1 07 of the tenth em- 
bodiment. Once the program started in Step S131, the 
information providing engine 107 examines a current 



state of the television application program at specified 
time intervals (Step SI 32). If no TV program is viewed 
(Step SI 33), the engine 1 07 returns to Step SI 32. When 
a TV program is watched (Step SI 33), the engine 107 

5 refers to the EPG and extracts therefrohi a keyword (e. 
g., title) of the program being viewed by the user (Step 
S134). The engine 1 07 transfers the obtained keyword 
to the action script execution engine 101 and then re- 
turns to Step S132. The action script may be selected 

10 depending on the keyword and the conversation script 
may be decided depending on the keyword. 
[0075] The above operation of the embodiment pro- 
vides such a particular effect that a conversation theme 
can be changed depending on a detected keyword. 

15 

[Eleventh Embodiment] 

[0076] Fig. 14 shows the fourth example of a display 
image of an agent, according to the eleventh embodi- 
ment of the present invention. When one of programs 
frequently viewed by the user and registered in the user 
preference database DB108 has not been watched by 
the user within the time interval of n days after the last 
watching date, the embodiment transfers the informa- 
tion to the action script execution engine 101 that in turn 
displays the agent on a display screen to execute the 
conversation script regarding the fact that the user has 
not watched the favorite program. For example, when 
the time interval from the last watching a soccer game 
broadcast exceeds 30 days, the agent is displayed to 
output a message "Why didn't you watch recent soccer 
game broadcasts? Your favorite team has lost 
games? ". This message can be outputted in the speech 
mode and the text mode at a time. 
[0077] Fig. 15 is a flowchart depicting the operation 
of the information providing engine 107 in the eleventh 
embodiment. Once the program started in Step SI 51, 
the infonnation providing engine 1 07 examines program 
information registered In the preference database 
DB108 at specified time intervals (Step SI 52). In Step 
SI 53. the engine 1 07 examines whether any one of the 
programs registered in the user preference database 
DB1 08 has an interval of more than n days from the last 
watching date. If not, the engine 1 07 returns to Step 
SI 52. When there is found a program having the non- 
watched period exceeding n days, the engine 107 
transfers the detected program information to the action 
script execution engine 101 and then returns to Step 
SI 52. The action script to be executed is decided in ac- 
cord with the non-watched period. The conversation 
content may be changed depending on the non- 
watched period (time Interval). 

[0078] As the result of the above operation of the em- 
bodiment, a theme of the conversation can be selected 
in accord with the non-watched period. 
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[Twelfth Embodiment] 

[0079] Fig. 1 6 shows the fifth example of a display im- 
age of an agent, according to the twelfth embodiment of 
the present invention. The agent Interface device iden- 
tifies a user preferred program genre by calculating a 
sum of viewing frequencies for each genre from usage 
history information stored in the preference database 
DB1 08, recognizes the time of starting the television ap- 
plication program by the timer 23, selects the preferred 
genre programs being currently broadcast or to be 
broadcast after now from a TV program guide obtained 
by the EPG acquisition means 1 06 and displays a table 
of genre-based preferred programs on the display 
screen. For example, if the user has frequently viewed 
broadcasts of soccer games, sports Is Identified as the 
preferred genre and a table of programs of the sport 
genre is displayed. 

[0080] Fig. 17 is a flowchart depicting the operation 
of the inf omriation providing engine 1 07 of the twelfth 
embodiment. Once the program started in Step SI 71, 
the information providing engine 107 extracts the view- 
ing frequency data for each genre from the usage history 
infonnation stored in the user preference database 
DB108 at specified time intervals (Step 8172). Namely, 
the engine 81 07 identifies the genre of programs having 
the highest viewing frequency (Step 81 73). The engine 
107 examines whether the television application pro- 
gram is driven or not (Step S 1 74). If not, the engine 1 07 
returns to Step SI 72. If the television application pro- 
gram was driven, the engine 107 recognizes, from the 
timer 112, the time of starting the television application 
program. The engine 107 obtains a table of programs 
being currently broadcast or to be broadcast from TV 
program guides acquired by the EPG acquisition means 
1 06 (Step 81 76) and extracts from the above table only 
programs of the genre decided in Step SI 73. Finally, the 
engine 1 07 displays a table of selected programs on the 
display screen (Step SI 78) and then retums to Step 
S172. 

[0081 ] As the resu It of the above operation of the em- 
bodiment, a table of preferred genre programs can be 
prepared based on the usage history and displayed on 
the display screen immediately after starting the televi- 
sion application program. 

[Thirteenth Embodiment] 

[0082] Fig. 1 B shows the sixth example of a display 
image of an agent, according to of the thirteenth embod- 
iment of the present invention. Once the user activated 
an Internet browser application program to browse web 
sites, this embodiment monitors the duration of connec- 
tion of the user's communication line to the Internet by 
the timer 112 and, when n minutes elapsed, displays the 
agent on the display screen to inform the user that the 
communication line is stili connected to the Internet. 
For example, when 120 minutes elapsed after starting 



the Internet browsing application and connecting the 
communication line to the Internet, the agent is dis- 
played on the display screen to output a warning mes- 
sage "Your line is always busy. The connection is still 
5 live". This message may be outputted vocally and visu- 
ally at a time. 

[0083] Fig. 19 is a flowchart depicting the operation 
of an agent interface device according to the thirteenth 
embodiment of the present invention. Once the program 

10 started in Step SI 91 . the agent interface device exam- 
ines whether the internet browser application program 
is driven (Step SI 92). If not, the device returns to Step 
S192. If the program was driven, the device examines 
whether the communication line is connected to the In- 

15 ternet (Step SI 93). If not, the device returns to Step 
S1 92. When the communication line is connected to the 
Internet, the device counts the connection period by the 
timer 112. The device examines whetherthe connection 
period reaches n minutes (Step SI 95). If not, the device 

20 returns to Step S192. If n minutes elapsed, the device 
transfers an action script descriptive of "the communi- 
cation line being left as connected to the Internet" to the 
action script execution engine 101. The time setting can 
be freely changed by the user. The device then returns 

25 to Step SI 92. 

[0084] As the result of the above operation of the em- 
bodiment, the agent can inform the user of the commu- 
nication line being left as connected to the internet when 
n minutes elapsed after driving the internet browser ap- 

3o plication program. 

[Fourteenth Embodiment] 

[0085] Fig. 20 shows the seventh example of a display 

35 image of an agent, according to the fourteenth embod- 
iment of the present invention. When the user activated 
a plurality of application programs and the remaining ca- 
pacity of the system resource was reduced to n%, this 
embodiment displays the agent on the display screen to 

4o inform the user that the operation will be unstable due 
to shortage of the system resource. For example, when 
the remaining capacity of the system resource was re- 
duced to 20%, the agent appears on the display screen 
and outputs a warning message 'The system is in an 

-♦5 unstable state. Please finish any currently unused pro- 
gram". This message can be outputted in both the 
speech mode and the text mode at a lime. 
[0086] Fig. 21 is a flowchart depicting the operation 
of an agent interface device according to the fourteenth 

50 embodiment of the present invention. When the pro- 
gram started In Step S211, the agent Interface device 
examines the remaining capacity of the system re- 
source (Step 321 2) and decides whether the remaining 
resource capacity is no more than n %. In Step S213, if 

55 it is more than n %, the device retums to Step S212. If 
it is no more than n %, the device transfers action script 
describing "shortage of the system resource* to the ac- 
tion script execution engine 101 (Step S214). 
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[0087] As the result of the above operation of the em- 
bodiment, the agent can inform the user of the shortage 
of the system resource when the remaining resource ca- 
pacity Is reduced to n %. 

[0088] It should be understood that the foregoing re- 
lates to only prefen'ed embodiments of the invention, 
and that the invention is not limited to the specific em- 
bodiments thereof except as defined In the appended 
claims. 

[0089] The agent interface device according to the 
present invention can offer the following advantages: 
[0090] The user can operate a variety of applications 
through interactions with the personified agent inter- 
face. Namely, the agent interface device may serves as 
a unified interface for a variety of applications. 
[0091] The user may not loss a chance of viewing a 
favorite television program because the agent previous- 
ly informs him/her that the program will be broadcasted. 
[0092] The user can enjoy a conversation with the 
agent who can speak about the television program the 
user has just viewed. 

[0093] Detailed data related to the television program 
that the user just has viewed can be obtained from web 
sites by browsing, which is realized through the interac- 
tion with the agent interface. 

[0094] The agent interface can prepares a table of 
preferred genre programs based on the usage history 
data stored in the user preference database and pre- 
sented it on the display screen, wherefrom the user can 
know new interesting programs that he/she has not 
viewed. 

[0095] When the communication line was left as con- 
nected to the Internet for a specified time, the agent in- 
terface informs the user of the connected state of the 
communication line. The user can cut off the communi- 
cation line in due time. 

[0096] The agent interface informs the user that the 
remaining capacity of the system resource may be short 
for the current applications. This enables the user to 
take suitable measures to avoid the unstable operation 
of the system. 



Ciainns 

1 . An agent interface device for displaying an artificial 
agent for providing information by the agent's action 
and speech, comprising: 

an agent interface (100) having an agent dis- 
play control means (1 09) for controlling the dis- 
play and action of the agent, a vok:e output 
means (110) for generating a speech of the 
agent, and an input means (111) for input by a 
user, 

an action script execution engine (1 01 ) for driv- 
ing the agent interface into actual action ac- 
cording to an action script describing means to 



control the agent interface, 
a retrieval engine (1 02) for retrieving an adapt- 
ed action script from an action script database 
and transferring the action script to the action 
script executing engine, 

an application program interface (105) for con- 
trolling an application program, 
an electronic program guide information acquir- 
ing means (106) for acquiring an electronic 
broadcast program guide, 
a preference database (1 08) for analyzing user 
preferences from Information provided by the 
electronic program guide information acquiring 
means and the application program interface 
and storing analysis results therein, 
an information providing engine (107) for pro- 
viding information to a user according to infor- 
mation stored in the preference database, and 
a timer (112) for setting a time for providing in- 
formation to the user by the information provid- 
ing engine. 

2. An agent interface device as defined in claim 1, 
characterized in that it is further provided with 
means capable of deciding a user preferred televi- 
sion program from a user usage history of a televi- 
sion application controllable by the application pro- 
gram interface (105) according to information from 
the electronic program guide information acquiring 
means (i06) and Information from the preference 
database (108), of displaying the agent at a speci- 
fied time (n minute) before a start time of the user 
preferred TV program and, of causing the agent to 
have a conversation with the user about the user 
preferred TV program by a speech output through 
the voice outputting means. 

3. An agent interface device as defined in claim 1, 
characterized in that it is further provided with 
means capable of deciding a user preferred genre 
from a user usage history of a television application 
controllable by the application program interface 
(105) according to information from the electronic 
program guide information acquiring means (106) 
and Infomnation from the preference database 
(1 08), of displaying the agent at a specified time (n 
minutes) before a start time of the user preferred 
genre TV program, and of causing the agent to have 
a conversation with the user about the user pre- 
ferred genre by a speech through the voice output- 
ting means (110). 

4. An agent interface device as defined in claim 1, 
characterized in that it is further provided with 
means capable of displaying the agent immediately 
after the end of a TV program of a television appli- 
cation controllable by the application program inter- 
face (105) and of causing the agent to have a con- 
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versation with the user about the just viewed pro- 
gram. 

5. An agent interface device as defined in claim 4, 
characterized in that the agent is displayed imme- 
diately after ending a television application control- 
lable by the application program interface (105) and 
an internet retrieval means for retrieving information 
on the Internet is provided to retrieve information 
from a web site relevant to the just viewed program. 

6. An agent interface device as defined in claim 4. 
characterized in that it is further provided with 
means capable of displaying the agent immediately 
after ending a video recording device controllable 
by the application program interface (105) and of 
making conversation with the user about a just 
viewed video program. 

7. An agent interface device as defined in claim 5. 
characterized in that the agent is displayed imme- 
diately after ending a video recording device con- 
trollable by the application program interface (1 05) 
and an internet retrieval means for retrieving infor- 
mation on the Internet is further provided to retrieve 
information from a web site relevant to a just viewed 
video program. 

8. An agent Interface device as a defined in claim 4, 
characterized in that it is provided with means ca- 
pable of displaying the agent immediately after end- 
ing an Internet browser application program control- 
lable by the application program interface (105) and 
of causing the agent to make conversation with the 
user about a just browsed web site. 

9. An agent interface device as defined in claim 5, 
characterized in that the agent is displayed imme- 
diately after ending an internet browser application 
program controllable by the application program in- 
terface (1 05) and an intemet retrieval means for re- 
trieving infonmation from a web site on an internet 
is provided to browse a web site relevant to a just 
browsed site. 

10. An agent interface device as defined in claim 1, 
characterized in that it is capable of changing a 
theme of a conversation in accordance with a key- 
word of a program displayed by a television appli- 
cation controllable by the application program inter- 
face (105). 

11. An agent interface device as defined in claim 1, 
characterized in that It is capable of changing a 
theme of a conversation according to time interval 
information obtainable from a usage history of a tel- 
evision application controllable by the application 
program interface (105). 



18 

12. An agent interface device as defined in claim 1, 
characterized in that a table of frequently watched 
genre programs is prepared on the basis of a usage 
history of a television application controllable by the 

s application program interface (105) and displayed 

when driving a television application controllable by 
the application program interface. 

13. An agent interface device as defined in claim 1, 
10 characterized in that the agent is capable of In- 
forming the user of a communication line still con- 
nected to the Intemet when n minutes elapsed after 
driving an intemet browser application program 
controllable by the application program interface 

15 (105). 

14. An agent interface device as defined in claim 1, 
characterized in that the agent is capable of in- 
forming the user of shortage of a system resource 
when a resource of the system runs short (It be- 
comes n %). 
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